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DETAILED ACTION 



1. 



Claims 1-44 are presented for examination. 



Election/Restrictions 



2. Restriction to one of the following invention is required under 35 U.S.C. 121 
Group I. Claims 1-42 are drawn to searching multimedia objects using a 

keyword and content features, which is classified in Class 707, subclass 3. 

Group II. Claims 43-44 are drawn to creating a user interface to allow the user to 
browse and sending feedback to the system, which is classified in Class 345, subclass 
700. 

3. Inventions I and II are related as subcombinations disclosed as usable together a 
single combination. The subcombinations are distinct from each other if they are shown 
to be separately usable. In the instant case, invention I is drawn to searching and 
retrieving the multimedia by a keyword and content features. The created user interface 
in the invention II is utilized to allow the user to browse the multimedia objects and send 
user's feedback to the system relating the multimedia objects. See M.P.E.P 806.05(d) 
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4. Because of the inventions are distinct for the given reasons and have acquired in 
a separate status in the art as show by their different classification, restriction for 
examination purposes as indicated is proper. 

5. Because these inventions are distinct for the reasons given above and the 
search required for Group I is not required Group II, restriction for examination purpose 
as indicated is proper. 

6. During the telephone conversation with Mr. Lewis C. Lee, Reg. No. 34,656 on 
October 29, 2002, a provisional election was made to prosecute the invention of Group 
I, claims 1-42. Affirmation of this election must be made by applicant in replying to this 
office action. Claims 42-44 have been withdrawn from further consideration by the 
examiner, 37 CFR 1.142(b), as being drawn to a non-elected invention. 
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Claim Rejections - 35 USC § 103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 1-42 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Amano et al. (A flexible content-based image retrieval system with combined scene 
description word, page 201-208, June 17-23, 1996, Multimedia Computing and System) 
in view of Graham et al. (US. Patent No. 6,369,81 1 ). 

Regarding on claims 1 and 9, teaches a method comprising: 
identifying, in response to a search query, first multimedia objects having an 
associated keyword that matches a keyword in the search query and second multimedia 
objects that have content features similar to those of the first multimedia objects; 

presenting the first and second multimedia objects to a user [fig. 4 and 5, right 
column, page 204]; 
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Amano does not explicitly teach monitoring feedback from the user as to which of 
the first and second multimedia objects are relevant to the search query; and 
annotating one or more of the multimedia objects, which are deemed relevant by the 
user, with keyword. However, Graham teaches, "the modification may occur in the 
background or may involve explicit user feedback input. The locations of concept if 
interest determined by pattern identification stage 620 are monitored by profile updating 
stage 634. Profile updating stage 624 notes the proximity of other keywords and key 
phrase within each analyzed document to the location of concepts of interest. If 
particular keyword or key phrases are always near a concept of interest, the structure 
and contents of belief system 700 are updated in the background without the user input 
by profile updating data. This could mean changing probability values, introducing a 
new connection between a subconcept and concept, or introducing a new keyword or 
key phrase" [col. 7, lines 52-65]. This teaches monitoring feedback and annotation 
process. Therefore, it would have been obvious to one ordinary skill in the art at the 
time of the invention was made to modify the teaching of Graham in to Amano because 
by utilizing monitoring user's feedback and an automatic annotator would allow the 
system to finds the keyword that best described the multimedia object and allowing the 
user retrieves more precise result when searching. 



# 
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Regarding on claim 2 f Graham teaches maintaining associations between the 
keywords and the multimedia objects, the associations being weighted to indicate how 
relevant the keywords are to the multimedia objects [col. 8, lines 2-6]. 

Adjusting the weights of the associations based on the user's feedback [col. 8, 
lines 10-13]. 

Regarding on claim 3, Graham teaches the adjusting comprises increasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed relevant by the user [col. 8, lines 6-10]. 



Regarding on claim 4, Graham teaches the adjusting comprises decreasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed irrelevant by the user [col. 8, lines 10-13]. 
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Regarding on claim 5, Graham does not explicitly teach removing the keyword 
from the particular multimedia object in an event that the weight is less than a threshold 
value. However, Graham teaches, "the user adds to the list by selecting the keyword 
add button 828 which causes display of a dialog box (not shown) for entering the new 
keyword or key phrases. The user deletes a keyword or key phrase by selecting it and 
then selecting a keyword delete button 830" [col. 8, lines 54-58]. This teaches that the 
user have the ability to removing the keyword or key phrases. Therefore, it would have 
been obvious to one ordinary skill in the art at the time of the invention was made to 
modify the teaching of Graham from the manual to an automatic annotation to allow the 
user less skill of the computer to effectively get the right results. 

Regarding on claim 6, Graham teaches training how the first and second 
multimedia objects are identified based on the user's feedback [col. 7, lines 59-65]. 

Regarding on claim 7, Graham teaches refining the search to identify additional 
multimedia objects that contain content features similar to those of the multimedia 
objects indicated by the user as being relevant [col. 7, lines 3-5]. 

Regarding on claim 8, Amano teaches the multimedia objects comprise one of 
digital images, video objects, and audio objects [fig. 5]. 
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Regarding on claims 10 and 17, Amano teaches a method comprising: 
Iteratively retrieving multimedia objects from a database [page 204, right column, 
fig. 5]. Amano does not explicitly teaches monitoring feedback from a user to whether 
the multimedia objects are relevant to a keyword in a search query; and annotating the 
multimedia objects based on the user's feedback. However, Graham teaches, "the 
modification may occur in the background or may involve explicit user feedback input. 
The locations of concept if interest determined by pattern identification stage 620 are 
monitored by profile updating stage 634. Profile updating stage 624 notes the proximity 
of other keywords and key phrase within each analyzed document to the location of 
concepts of interest. If particular keyword or key phrases are always near a concept of 
interest, the structure and contents of belief system 700 are updated in the background 
without the user input by profile updating data. This could mean changing probability 
values, introducing a new connection between a subconcept and concept, or 
introducing a new keyword or key phrase" [col. 7, lines 52-65]. This teaches monitoring 
feedback from the user and annotated the multimedia with user's feedback. Therefore, 
it would have been obvious to one ordinary skill in the art at the time of the invention 
was made to modify the teaching of Graham in to Amano because by utilizing the 
monitoring the user's feedback and an automatic annotator would allow the system to 
analyze the user's feedback and change the best describing keyword to the multimedia 
objects. 
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Regarding on claim 1 1 , Amano teaches the retrieving comprises using content- 
based information retrieval to retrieve the multimedia objects [page 204, left column, 
lines 20-21]. 

Regarding on claim 12, Amano teaches the retrieving comprises using both 
content-based information retrieval and semantic-based information retrieval to retrieve 
the multimedia objects [page 204, left column, lines 31-33]. 

Regarding on claim 13, Graham teaches the monitoring comprises monitoring 
both feature-based relevance feedback and semantic-based relevance feedback [col. 7, 
lines 54-56]. 

Regarding on claim 14, Graham teaches the annotating is hidden from the user 
[col. 7, lines 52-54]. 
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Regarding on claim 15, Graham teaches the annotating comprises: 

in an event that a particular multimedia object is deemed relevant by the user 

and is not yet annotated with the keyword, adding the keyword to the particular 

multimedia object [col. 8, lines 6-10]; and 

in an event that the particular multimedia object is deemed relevant by the user 

and is already annotated with the keyword, weakening an association between the 

keyword and the particular multimedia object [col. 8, lines 10-14]. 

Regarding on claim 16, Graham teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, weakening an association between the 

keyword and the particular multimedia object [col. 8, lines 10-14], 
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Regarding on claims 18 and 24, Amano teaches method comprising: 

retrieving multimedia objects according to a content-based retrieval process 
[page 204, left column, lines 19-20]; 

presenting the multimedia object to a user [page 204, right column, fig. 5]; 

Amano does not explicitly teach monitoring feedback from the user as to which 
of the multimedia objects are relevant; and annotating one or more of the multimedia 
objects based on the user's feedback. However, Graham teaches, "the modification 
may occur in the background or may involve explicit user feedback input. The locations 
of concept if interest determined by pattern identification stage 620 are monitored by 
profile updating stage 634. Profile updating stage 624 notes the proximity of other 
keywords and key phrase within each analyzed document to the location of concepts of 
interest. If particular keyword or key phrases are always near a concept of interest, the 
structure and contents of belief system 700 are updated in the background without the 
user input by profile updating data. This could mean changing probability values, 
introducing a new connection between a subconcept and concept, or introducing a new 
keyword or key phrase" [col. 7, lines 52-65]. This teaches monitoring the feedback from 
the user and annotating by the user's feedback. Therefore, it would have been obvious 
to one ordinary skill in the art at the time of the invention was made to modify the 
teaching of Graham in to Amano because by utilizing monitoring user's feedback and an 
automatic annotator would allow the next user to retrieve more precise search result. 
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Regarding on claim 19, Graham teaches the monitoring comprises monitoring 
both feature-based relevance feedback and semantic-based relevance feedback [col. 7, 
lines 54-56]. 

Regarding on claim 20, Graham teaches the annotating is hidden from the user 
[col. 7, lines 52-54]. 

Regarding on claim 21, Graham teaches the annotating comprise: 

In an event that a particular multimedia object is deemed relevant by the user 

and not yet annotated with the keyword, adding the keyword to the particular multimedia 

object [col. 7, lines 64-65]; and 

In an event that the particular multimedia object is deemed relevant by the user 

and is already annotated with the keyword, strengthening an association between the 

keyword and the particular object [col. 8, lines 6-10]. 

Regarding on claim 22, Graham teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, weakening an association between the 

keyword and the particular multimedia object [col. 8, lines 10-14]. 
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Regarding on claim 23, Graham teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, removing the keyword from the particular 

multimedia object [col. 7, lines 64-65]. 
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Regarding on claims 25 and 31 , Amano teaches a method comprising: 

maintaining association between keywords and multimedia objects, the 
associations being weight to indicate how relevant the keywords are to the multimedia 
objects [page 202, left column, fig. 1]; and 

retrieving a set of one or more multimedia objects for presentation to a user 
[page 204, right column, fig. 5]. 

Amano does not explicitly teach monitoring feedback from the user as to which of 
the multimedia objects are relevant; and adjusting the weights of the associations based 
on the user's feedback. However, Graham teaches, "the modification may occur in the 
background or may involve explicit user feedback input. The locations of concept if 
interest determined by pattern identification stage 620 are monitored by profile updating 
stage 634. Profile updating stage 624 notes the proximity of other keywords and key 
phrase within each analyzed document to the location of concepts of interest. If 
particular keyword or key phrases are always near a concept of interest, the structure 
and contents of belief system 700 are updated in the background without the user input 
by profile updating data. This could mean changing probability values, introducing a 
new connection between a sub concept and concept, or introducing a new keyword or 
key phrase" [col. 7, lines 52-65]. This teaches monitoring feedback that the user input 
after they review the objects. 
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In addition, Graham also teaches, "if, on the other hand, user 504 indicates the 
selected keywords or key phrases to be of little interest, the probability values 
connecting these keywords or key phrases to the concept decrease" [col. 8, lines 10- 
14]. This teaches the adjusting the weight of the association between the keyword or 
key phrases based on the user. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to modify the teaching of Graham in 
to Amano because by utilizing the monitoring the user's feedback and adjusting 
mechanism to allow the system to change the new keyword that best describing the 
multimedia objects. 

Regarding on claim 26, Amano teaches the retrieving comprises using content- 
based information retrieval to retrieve the multimedia objects [page 204, left column, 
lines 19-20]. 

Regarding on claim 27, Amano teaches the retrieving comprises using content- 
based information retrieval and semantic-based information retrieval to retrieve the 
multimedia objects [page 204, left column, lines 31-34]. 

Regarding on claim 28, Amano teaches the monitoring comprises capturing both 
feature-based relevance feedback and semantic-based relevance feedback [col. 7, lines 
54-56], 
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Regarding on claim 29, Graham teaches the adjusting comprises increasing the 
weights of the association between the keywords and the multimedia objects that are 
deemed relevant by the user [col. 8, lines 6-10]. 

Regarding on claim 30, Graham teaches adjusting comprises decreasing the 
weights of the association between the keywords and the multimedia objects that are 
deemed irrelevant by the user [col. 8, lines 10-14]. 
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Regarding on claim 32, Amano teaches a system comprising: 
An information retrieval unit to retrieve multimedia objects from a database based 
on a search query [page 204, left column, lines 20-21]; and 

Amano does not explicitly teaches relevance feedback unit to capture a user's 
feedback as to whether the multimedia objects are relevant to the search query; and an 
annotation unit to annotate the multimedia objects based on the user's feedback. 
However, Graham teaches, "the modification may occur in the background or may 
involve explicit user feedback input. The locations of concept if interest determined by 
pattern identification stage 620 are monitored by profile updating stage 634. Profile 
updating stage 624 notes the proximity of other keywords and key phrase within each 
analyzed document to the location of concepts of interest. If particular keyword or key 
phrases are always near a concept of interest, the structure and contents of belief 
system 700 are updated in the background without the user input by profile updating 
data. This could mean changing probability values, introducing a new connection 
between a subconcept and concept, or introducing a new keyword or key phrase" [col. 
7, lines 52-65]. This teaches monitoring unit analyzes the user's feedback and 
annotation unit change the new keyword to the objects. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention was made to modify 
the teaching of Graham in to Amano because by utilizing the monitor unit and 
annotation units would allow the system to analyzes the user's feedback and change 
the new keyword that introduce by the user to the multimedia objects latter allow the 
other users to search and retrieve more precise results. 
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Regarding on claim 33, Amano teaches the search query comprises a keyword- 
based search query having one or more keywords [page 203, right column, lines 28-35]. 

Regarding on claim 34, Amano teaches the search query comprises a content- 
based search query having one or more content features [page 202, left column, lines 
15-18]. 

Regarding on claim 35, Amano teaches the information retrieval unit employs 
both content-based (blue) information and semantic based information retrieval (lower) 
[page 204, left column, lines 32-34]. 

Regarding on claim 36, Amano teaches the information retrieval unit comprises: 
A query handler to handle both keyword-based queries having one or more 
search keywords and content-based queries having one or more content features of a 
multimedia object [page 205, left column, lines 32-36]; and 

A feature and semantic matcher to identify at least one of (1 ) first multimedia 
object having keywords that match the search keywords from a keyword-based query, 
and (2) second multimedia objects having content features similar to the content 
features of a content-based query [page 205, left column, lines 38-45 and right column, 
lines 1-16]. 
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Regarding on claim 37, Graham teaches the relevance feedback unit employs 
both feature-based relevance feedback and semantic-based relevance feedback [col. 7, 
lines 54-56]. 

Regarding on claim 38, Graham teaches the search query comprise a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed relevant by the user and is not yet annotated with 
keyword, the annotation units adds the keyword to the particular multimedia object [col. 
7, lines 54-65]. 

Regarding on claim 39, Graham teaches the search query comprises a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed relevant by the user and is already annotated with the 
keyword, the annotation unit strengthens an association between the keyword and the 
particular multimedia object [col. 8, lines 6-10]. 

Regarding on claim 40, Graham teaches the search query comprises a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed irrelevant by the user and is already annotated with the 
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keyword, weakening an association between the keyword and the particular multimedia 
object [col. 8, lines 10-14]. 

Regarding on claim 41, Graham teaches the search query comprises a keyword- 
based search query having at least one keyword [page 203, right column, lines 28-35]; 
and in an event that a particular multimedia object is deemed irrelevant by the user and 
is already annotated with the keyword, removing the keyword from the particular 
multimedia object [col. 7, lines 64-66] 



Regarding on claim 42, Graham teaches the relevance feedback unit comprises 
a feedback analyzer to train the system based on the user's feedback [col. 7, lines 61- 
65]. 
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Conclusion 



8. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 



us. 


Patent No. 6,041,335 


Merrit et al. 


March 21, 2000 


us. 


Patent No. 6,301,586 


Yang et al. 


Oct 9, 2001 


us. 


Patent No. 6,105,055 


Pizano et al. 


Aug 15, 2000 


us. 


Patent No. 6,408,293 


Aggarwal et al. 


June 16, 2002 
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1949 or via e-mail BaoquocN.To@uspto.gov. The examiner can normally be reached 
on Monday-Friday: 8:00 AM -4:30 PM, EST. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kim Y. Vu can be reached at (703) 305-4393. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the receptionist whose telephone number is (703) 305- 
3900. 

Any response to this action should be mailed to: 
Commissioner of Patents and Trademarks 
Washington, D.C. 20231. 

The fax numbers for the organization where this application or proceeding is 
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• (703) 746-7238 [After Final Communication}] 

• (703) 746-7239 [Official Communication] 

• (703) 746-7240 [Non-Official Communication] 
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Crystal Park II 
2121 Crystal Drive 
Arlington, VA 22202 
Fourth Floor (Receptionist). 




Baoquoc N. To 
October 31, 2002 



